Beyond Translation Memories: finding similar documents in comparable corpora

This paper presents our most recent research in the context of TTC, an EU funded research project, on using the Web to retrieve terminologically rich texts in a specific domain, and to find similar documents in such comparable corpora. The aim of this work is to provide tools for semi-automatic construction of bilingual term lists. 1 Parallel and comparable corpora Re-use of existing translations lies at the centre of translation practice as we know it now. Translation Memories fuel the daily activities of translators, while Machine Translation engines build their translation models from collections of parallel texts (this applies to both Statistical MT, like Google, and to traditional rule-based MT systems enriched with statistics, like Systran). However, many more texts are produced in each language on a daily basis than translated by professional translators. Even when translated texts exist, they are often not available in a given application or for a given translation task. There is a particular shortage of parallel texts in the areas undergoing recent developments. This leads to great interest in utilising comparable (=less parallel) resources for translation tasks. Informally, any collection of texts in two languages can be positioned on a cline from ‘fully parallel’ to ‘unrelated’ with several options in between: noisy parallel texts: such texts introduce minor language-specific adaptations, e.g., an example for searching New York in the OpenOffice manual might be replaced with北京 (‘Beijing’) in its Chinese translation; strongly comparable texts: such texts are not designed to be translations, while they are still devoted to the same narrow topic and their producers are aware of a related text in a different language, e.g., interlinked wikipedia articles or news items concerning exactly the same specific event in different languages; weakly comparable texts: such texts are devoted to the same topic, but they are produced completely independently, e.g., English and Chinese textbooks on designing the wind turbines, or parliamentary debates on health care from the Bundestag, the House of Commons and the Russian Duma. This paper reports our experience from the TTC project (Translation, Terminology and Comparable Corpora),1 which is aimed at developing tools for (1) collecting comparable corpora from the Web, (2) http://ttc-project.eu/